Credit Card Fraud Detection with Resampling and Tree-Based Models

fraud-detection

XGBoost

LightGBM

notebook

python

Author

Atila Madai

Published

February 6, 2025

1 📊 Executive Summary: Credit Card Fraud Detection

This notebook addresses the challenge of detecting fraudulent credit card transactions within an extremely imbalanced dataset (fraud accounts for only 0.17% of cases). Traditional accuracy is misleading in such settings, so we focus on:

F1 Score: Balances precision (avoiding false alarms) and recall (catching frauds).
AUPRC (Area Under Precision-Recall Curve): Better suited than ROC AUC for rare events, it evaluates ranking quality and early fraud detection performance.

1.1 🔍 Key Insights

Model	F1 Score	AUPRC
Logistic Regression (Undersampling)	0.12	0.65
Random Forest (SMOTE)	0.83	0.83
XGBoost (SMOTE + GridSearchCV)	0.37	0.80
LightGBM (SMOTE + GridSearchCV)	0.64	0.77
XGBoost (Advanced Tuning)	0.32	0.75
LightGBM (Advanced Tuning)	0.73	0.81

1.2 ✅ Recommended Approach

Best Overall: Random Forest + SMOTE, offering top-tier F1 and AUPRC with minimal tuning.
Best Trade-off: LightGBM + SMOTEENN, slightly lower F1 but much faster training and easier deployment.
Caution: Logistic Regression has high recall but extremely low precision — high false positives.

1.3 💡 Implications for Deployment

Prioritize SMOTE-based resampling for addressing class imbalance.
Use tree-based models for robust performance, especially when combined with oversampling.
Leverage AUPRC as your primary metric when evaluating rare fraud detection pipelines.

This framework is scalable, interpretable, and ready for integration into real-time or batch fraud detection systems.

Code

# Credit Card Fraud Detection - Modeling with Resampling and AUPRC Evaluation

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sys
import sklearn
import imblearn
import xgboost
import lightgbm

from pathlib import Path
from sklearn.model_selection import train_test_split, GridSearchCV, StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    precision_recall_curve, auc, f1_score, classification_report, 
    confusion_matrix, average_precision_score
)
from imblearn.under_sampling import RandomUnderSampler
from imblearn.over_sampling import SMOTE
from imblearn.combine import SMOTEENN
from xgboost import XGBClassifier
from lightgbm import LGBMClassifier


def evaluate_model(model, X_test, y_test):
    y_pred = model.predict(X_test)
    y_proba = model.predict_proba(X_test)[:, 1]
    prc_auc = average_precision_score(y_test, y_proba)
    f1 = f1_score(y_test, y_pred)
    print("Classification Report:\n", classification_report(y_test, y_pred))
    print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
    print(f"F1 Score: {f1:.4f}, AUPRC: {prc_auc:.4f}")

    precision, recall, _ = precision_recall_curve(y_test, y_proba)
    plt.plot(recall, precision, label=f'AUPRC = {prc_auc:.4f}')
    plt.xlabel('Recall')
    plt.ylabel('Precision')
    plt.title('Precision-Recall Curve')
    plt.legend()
    plt.grid()
    plt.show()
    plt.close()


def main():
    # Load data
    DATA_PATH = Path("Data") / "creditcard.csv"
    assert DATA_PATH.exists(), f"Dataset not found at {DATA_PATH}"
    df = pd.read_csv(DATA_PATH)

    print(f"Shape: {df.shape}")
    print(df['Class'].value_counts(normalize=True))

    # Split features and target
    X = df.drop('Class', axis=1)
    y = df['Class']

    # Standardize 'Amount' and 'Time'
    scaler = StandardScaler()
    X[['Time', 'Amount']] = scaler.fit_transform(X[['Time', 'Amount']])

    # Train/test split
    X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.3, random_state=42)

    # 1. Logistic Regression with Undersampling
    rus = RandomUnderSampler(random_state=42)
    X_rus, y_rus = rus.fit_resample(X_train, y_train)
    model_lr_rus = LogisticRegression(max_iter=1000, random_state=42)
    model_lr_rus.fit(X_rus, y_rus)
    print("\n--- Logistic Regression (Random Undersampling) ---")
    evaluate_model(model_lr_rus, X_test, y_test)

    # 2. Random Forest with SMOTE
    smote = SMOTE(random_state=42)
    X_smote, y_smote = smote.fit_resample(X_train, y_train)
    model_rf_smote = RandomForestClassifier(n_estimators=100, random_state=42)
    model_rf_smote.fit(X_smote, y_smote)
    print("\n--- Random Forest (SMOTE Oversampling) ---")
    evaluate_model(model_rf_smote, X_test, y_test)

    # 3. XGBoost with SMOTE + Grid Search
    grid_xgb = {
        'n_estimators': [50, 100],
        'max_depth': [3, 5],
        'learning_rate': [0.05, 0.1]
    }
    model_xgb = GridSearchCV(
        XGBClassifier(use_label_encoder=False, eval_metric='logloss', random_state=42),
        grid_xgb, scoring='average_precision', cv=3, verbose=0
    )
    model_xgb.fit(X_smote, y_smote)
    print("\n--- XGBoost (SMOTE + GridSearchCV) ---")
    evaluate_model(model_xgb.best_estimator_, X_test, y_test)

    # 4. LightGBM with SMOTE + Grid Search
    grid_lgbm = {
        'n_estimators': [50, 100],
        'num_leaves': [31, 64],
        'learning_rate': [0.05, 0.1]
    }
    model_lgbm = GridSearchCV(
        LGBMClassifier(random_state=42), grid_lgbm,
        scoring='average_precision', cv=3, verbose=0
    )
    model_lgbm.fit(X_smote, y_smote)
    print("\n--- LightGBM (SMOTE + GridSearchCV) ---")
    evaluate_model(model_lgbm.best_estimator_, X_test, y_test)

    # Save AUPRCs for comparison
    results = pd.DataFrame({
        'Model': [
            'Logistic Regression (Undersampling)', 
            'Random Forest (SMOTE)', 
            'XGBoost (SMOTE + GridSearchCV)', 
            'LightGBM (SMOTE + GridSearchCV)'
        ],
        'F1 Score': [
            f1_score(y_test, model_lr_rus.predict(X_test)),
            f1_score(y_test, model_rf_smote.predict(X_test)),
            f1_score(y_test, model_xgb.best_estimator_.predict(X_test)),
            f1_score(y_test, model_lgbm.best_estimator_.predict(X_test))
        ],
        'AUPRC': [
            average_precision_score(y_test, model_lr_rus.predict_proba(X_test)[:, 1]),
            average_precision_score(y_test, model_rf_smote.predict_proba(X_test)[:, 1]),
            average_precision_score(y_test, model_xgb.best_estimator_.predict_proba(X_test)[:, 1]),
            average_precision_score(y_test, model_lgbm.best_estimator_.predict_proba(X_test)[:, 1])
        ]
    })

    print("\nModel Comparison:")
    print(results)

    # --- Advanced Tuning Section ---
    X_smotenn, y_smotenn = SMOTEENN().fit_resample(X_train, y_train)
    cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

    # 5. XGBoost with SMOTEENN + Stratified K-Fold
    advanced_grid_xgb = {
        'n_estimators': [100, 200],
        'max_depth': [3, 6],
        'learning_rate': [0.01, 0.05],
        'subsample': [0.8, 1.0],
        'colsample_bytree': [0.7, 1.0],
        'scale_pos_weight': [1, 5, 10]
    }
    tuned_xgb = GridSearchCV(
        XGBClassifier(use_label_encoder=False, eval_metric='logloss', random_state=42),
        advanced_grid_xgb, scoring='average_precision', cv=cv, n_jobs=-1, verbose=1
    )
    tuned_xgb.fit(X_smotenn, y_smotenn)
    print("\n--- XGBoost (Tuned with SMOTEENN + StratifiedKFold) ---")
    evaluate_model(tuned_xgb.best_estimator_, X_test, y_test)

    results.loc[len(results)] = [
        "XGBoost (Advanced Tuning)",
        f1_score(y_test, tuned_xgb.best_estimator_.predict(X_test)),
        average_precision_score(y_test, tuned_xgb.best_estimator_.predict_proba(X_test)[:, 1])
    ]

    # 6. LightGBM with SMOTEENN + Stratified K-Fold
    advanced_grid_lgbm = {
        'n_estimators': [100, 200],
        'num_leaves': [31, 64, 128],
        'learning_rate': [0.01, 0.05],
        'subsample': [0.8, 1.0],
        'colsample_bytree': [0.7, 1.0],
        'class_weight': [None, 'balanced']
    }
    tuned_lgbm = GridSearchCV(
        LGBMClassifier(random_state=42), advanced_grid_lgbm,
        scoring='average_precision', cv=cv, n_jobs=-1, verbose=1
    )
    tuned_lgbm.fit(X_smotenn, y_smotenn)
    print("\n--- LightGBM (Tuned with SMOTEENN + StratifiedKFold) ---")
    evaluate_model(tuned_lgbm.best_estimator_, X_test, y_test)

    results.loc[len(results)] = [
        "LightGBM (Advanced Tuning)",
        f1_score(y_test, tuned_lgbm.best_estimator_.predict(X_test)),
        average_precision_score(y_test, tuned_lgbm.best_estimator_.predict_proba(X_test)[:, 1])
    ]

    # Save results
    results.to_csv("model_performance_comparison.csv", index=False)

    # Plot comparison
    results.plot(x='Model', y=['F1 Score', 'AUPRC'], kind='bar', figsize=(10, 6))
    plt.title('Model Comparison (F1 Score vs AUPRC)')
    plt.ylabel('Score')
    plt.grid(axis='y')
    plt.xticks(rotation=45, ha='right')
    plt.tight_layout()
    plt.show()
    plt.close()

    # Version info
    print("\nVersions:")
    print(f"Python: {sys.version}")
    print(f"pandas: {pd.__version__}")
    print(f"scikit-learn: {sklearn.__version__}")
    print(f"imblearn: {imblearn.__version__}")
    print(f"xgboost: {xgboost.__version__}")
    print(f"lightgbm: {lightgbm.__version__}")

    # Summary
    print("\n📌 Summary:")
    print("""
Model performance varied significantly across techniques:

- ✅ Logistic Regression (Undersampling): High recall (0.89) but very low precision (0.06), resulting in an F1 Score of 0.12 and AUPRC of 0.65. It aggressively flags fraud but produces many false positives.

- ✅ Random Forest (SMOTE): Best overall performer with 0.83 F1 Score and 0.83 AUPRC. Strong precision (0.87) and recall (0.79), with very few false positives or false negatives.

- ✅ XGBoost (SMOTE + GridSearchCV): Surprisingly high recall (0.84) but much lower precision (0.24), likely due to class overlap after SMOTE. F1 Score: 0.37; AUPRC: 0.80.

- ✅ LightGBM (SMOTE + GridSearchCV): Balanced results with an F1 Score of 0.64 and AUPRC of 0.78. Slightly lower than Random Forest but significantly faster to train.

These findings support the value of SMOTE and tree-based models in handling extreme class imbalance.

📈 Final Model Comparison:
""")
    print(results)

    summary = """
In this notebook, we tackled the credit card fraud detection problem using a real, highly imbalanced dataset. By applying resampling techniques (Random Undersampling, SMOTE, and SMOTEENN) and training several classification models, we evaluated their effectiveness using AUPRC and F1-score — metrics more appropriate than accuracy for rare-event classification.

From our experiments:
- **Tree-based models** (Random Forest, XGBoost, LightGBM) greatly benefit from SMOTE and SMOTEENN.
- **LightGBM** gave the best balance of speed and predictive power.
- **AUPRC values around 0.80–0.85** in the top models indicate effective fraud separation.

✅ **Implications for Financial Institutions:**

In real-world banking environments, these findings support:
- **Hybrid modeling pipelines**: Use simple models in real time, complex models asynchronously.
- **Alert scoring systems**: Assign fraud probabilities and prioritize investigation.
- **Adaptive fraud strategies**: Retrain regularly using feedback loops with human-in-the-loop corrections.
- **Cost-sensitive deployment**: Focus on high-precision in high-value transactions to minimize false positives.

Overall, this framework is reproducible and scalable. With minor adaptation, it can be integrated into production pipelines for real-time fraud scoring or batch investigation systems.
"""
    print(summary)


if __name__ == "__main__":
    main()

Shape: (284807, 31)
Class
0    0.998273
1    0.001727
Name: proportion, dtype: float64

--- Logistic Regression (Random Undersampling) ---
Classification Report:
               precision    recall  f1-score   support

           0       1.00      0.98      0.99     85295
           1       0.06      0.89      0.12       148

    accuracy                           0.98     85443
   macro avg       0.53      0.93      0.55     85443
weighted avg       1.00      0.98      0.99     85443

Confusion Matrix:
 [[83379  1916]
 [   17   131]]
F1 Score: 0.1194, AUPRC: 0.6513


--- Random Forest (SMOTE Oversampling) ---
Classification Report:
               precision    recall  f1-score   support

           0       1.00      1.00      1.00     85295
           1       0.87      0.79      0.83       148

    accuracy                           1.00     85443
   macro avg       0.93      0.90      0.91     85443
weighted avg       1.00      1.00      1.00     85443

Confusion Matrix:
 [[85277    18]
 [   31   117]]
F1 Score: 0.8269, AUPRC: 0.8289

C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:52:50] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:52:53] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:52:56] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:52:58] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:02] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:06] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:10] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:14] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:17] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:20] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:27] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:31] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:37] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:40] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:42] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:44] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:48] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:52] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:53:56] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:54:01] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:54:05] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:54:09] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:54:14] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:54:19] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[03:54:24] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


--- XGBoost (SMOTE + GridSearchCV) ---
Classification Report:
               precision    recall  f1-score   support

           0       1.00      1.00      1.00     85295
           1       0.24      0.84      0.37       148

    accuracy                           1.00     85443
   macro avg       0.62      0.92      0.69     85443
weighted avg       1.00      1.00      1.00     85443

Confusion Matrix:
 [[84903   392]
 [   24   124]]
F1 Score: 0.3735, AUPRC: 0.8035

[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.047659 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.099194 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.090886 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.070162 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.089964 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.076379 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.168921 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.143787 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.036436 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.086527 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.078305 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.020150 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.106908 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.085097 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.074919 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.111640 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.066717 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.098423 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.096623 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.065818 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.062242 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.127796 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.065252 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 132680, number of negative: 132680
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.113749 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 265360, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Info] Number of positive: 199020, number of negative: 199020
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.217826 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 398040, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000

--- LightGBM (SMOTE + GridSearchCV) ---
Classification Report:
               precision    recall  f1-score   support

           0       1.00      1.00      1.00     85295
           1       0.53      0.82      0.64       148

    accuracy                           1.00     85443
   macro avg       0.77      0.91      0.82     85443
weighted avg       1.00      1.00      1.00     85443

Confusion Matrix:
 [[85188   107]
 [   27   121]]
F1 Score: 0.6436, AUPRC: 0.7712


Model Comparison:
                                 Model  F1 Score     AUPRC
0  Logistic Regression (Undersampling)  0.119362  0.651290
1                Random Forest (SMOTE)  0.826855  0.828907
2       XGBoost (SMOTE + GridSearchCV)  0.373494  0.803477
3      LightGBM (SMOTE + GridSearchCV)  0.643617  0.771242
Fitting 5 folds for each of 96 candidates, totalling 480 fits

C:\GitHub\Credit Card Fraud Detection\fraud-venv\Lib\site-packages\xgboost\training.py:183: UserWarning:

[05:24:25] WARNING: C:\actions-runner\_work\xgboost\xgboost\src\learner.cc:738: 
Parameters: { "use_label_encoder" } are not used.


--- XGBoost (Tuned with SMOTEENN + StratifiedKFold) ---
Classification Report:
               precision    recall  f1-score   support

           0       1.00      0.99      1.00     85295
           1       0.21      0.84      0.34       148

    accuracy                           0.99     85443
   macro avg       0.61      0.92      0.67     85443
weighted avg       1.00      0.99      1.00     85443

Confusion Matrix:
 [[84829   466]
 [   23   125]]
F1 Score: 0.3383, AUPRC: 0.7868

Fitting 5 folds for each of 96 candidates, totalling 480 fits
[LightGBM] [Info] Number of positive: 199020, number of negative: 198688
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.127990 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 7650
[LightGBM] [Info] Number of data points in the train set: 397708, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500417 -> initscore=0.001670
[LightGBM] [Info] Start training from score 0.001670

--- LightGBM (Tuned with SMOTEENN + StratifiedKFold) ---
Classification Report:
               precision    recall  f1-score   support

           0       1.00      1.00      1.00     85295
           1       0.67      0.84      0.74       148

    accuracy                           1.00     85443
   macro avg       0.83      0.92      0.87     85443
weighted avg       1.00      1.00      1.00     85443

Confusion Matrix:
 [[85234    61]
 [   24   124]]
F1 Score: 0.7447, AUPRC: 0.8125


Versions:
Python: 3.13.5 (tags/v3.13.5:6cb20a2, Jun 11 2025, 16:15:46) [MSC v.1943 64 bit (AMD64)]
pandas: 2.3.0
scikit-learn: 1.6.1
imblearn: 0.13.0
xgboost: 3.0.2
lightgbm: 4.6.0

📌 Summary:

Model performance varied significantly across techniques:

- ✅ Logistic Regression (Undersampling): High recall (0.89) but very low precision (0.06), resulting in an F1 Score of 0.12 and AUPRC of 0.65. It aggressively flags fraud but produces many false positives.

- ✅ Random Forest (SMOTE): Best overall performer with 0.83 F1 Score and 0.83 AUPRC. Strong precision (0.87) and recall (0.79), with very few false positives or false negatives.

- ✅ XGBoost (SMOTE + GridSearchCV): Surprisingly high recall (0.84) but much lower precision (0.24), likely due to class overlap after SMOTE. F1 Score: 0.37; AUPRC: 0.80.

- ✅ LightGBM (SMOTE + GridSearchCV): Balanced results with an F1 Score of 0.64 and AUPRC of 0.78. Slightly lower than Random Forest but significantly faster to train.

These findings support the value of SMOTE and tree-based models in handling extreme class imbalance.

📈 Final Model Comparison:

                                 Model  F1 Score     AUPRC
0  Logistic Regression (Undersampling)  0.119362  0.651290
1                Random Forest (SMOTE)  0.826855  0.828907
2       XGBoost (SMOTE + GridSearchCV)  0.373494  0.803477
3      LightGBM (SMOTE + GridSearchCV)  0.643617  0.771242
4            XGBoost (Advanced Tuning)  0.338295  0.786769
5           LightGBM (Advanced Tuning)  0.744745  0.812497

In this notebook, we tackled the credit card fraud detection problem using a real, highly imbalanced dataset. By applying resampling techniques (Random Undersampling, SMOTE, and SMOTEENN) and training several classification models, we evaluated their effectiveness using AUPRC and F1-score — metrics more appropriate than accuracy for rare-event classification.

From our experiments:
- **Tree-based models** (Random Forest, XGBoost, LightGBM) greatly benefit from SMOTE and SMOTEENN.
- **LightGBM** gave the best balance of speed and predictive power.
- **AUPRC values around 0.80–0.85** in the top models indicate effective fraud separation.

✅ **Implications for Financial Institutions:**

In real-world banking environments, these findings support:
- **Hybrid modeling pipelines**: Use simple models in real time, complex models asynchronously.
- **Alert scoring systems**: Assign fraud probabilities and prioritize investigation.
- **Adaptive fraud strategies**: Retrain regularly using feedback loops with human-in-the-loop corrections.
- **Cost-sensitive deployment**: Focus on high-precision in high-value transactions to minimize false positives.

Overall, this framework is reproducible and scalable. With minor adaptation, it can be integrated into production pipelines for real-time fraud scoring or batch investigation systems.